“PISA is not a football league table”

Enlarge

Katja Scharenberg is Professor of Research in Education and Socialization at LMU. Her research focuses on issues such as educational inequalities. In this interview, she discusses trends in the PISA results, school structures and the overall school experience.

Germany performed poorly in mathematics, reading and science in PISA 2022. You pointed out at the time that the downward trend in Germany had been evident for some time. What do you expect from the new results?

Katja Scharenberg: I would deliberately refrain from making any specific prediction before the results are published. It will be far more interesting to see whether the negative trend observed in previous PISA survey cycles continues, stabilizes or perhaps even reverses. In PISA 2022, Germany’s results in all three domains were the lowest ever recorded in PISA.

However, depending on the specific skill, this downward trend had already begun before the pandemic. So, we were not dealing exclusively with a pandemic effect. Rather than focusing on whether scores are a few PISA points higher or lower, I would look at three things: How is the average skill level developing? What proportion of young people fail to achieve basic levels of competence? And how do proficiency levels differ across various social groups?

The last point in particular is crucial for me, as it reveals whether potential improvements actually benefit all young people, or whether social inequalities in education are persisting or even widening.

Enlarge

If Germany performs poorly in PISA again, what can the results actually tell us about the causes? And does it even make sense to compare education systems based on a ranking system?

PISA is, first and foremost, a very good diagnostic study and a tool for empirical education monitoring. It allows us to see how skills are distributed, what trends emerge over time, and which individual and school-related characteristics are linked to differences in performance. PISA can highlight such correlations, but it cannot unequivocally show what is cause and what is effect. If, for example, pupils at schools with certain characteristics perform better, this does not necessarily mean that it is precisely these characteristics that have caused the better performance. I take a similarly cautious approach to rankings.

International performance comparisons are extremely valuable because they show that education systems achieve different results, and that high performance and low levels of social inequality can certainly go hand in hand. But PISA is not a football league table. For countries with similar results, minor differences in rankings are often not even statistically significant. Furthermore, education systems differ in terms of their social and institutional contexts. Rather than focusing on a single ranking, it is therefore more interesting to look at the trends and differences that emerge and what lessons we can learn from them for future research and education policy.

Enlarge

You have been conducting research into educational inequalities for many years. The PISA results repeatedly reveal a strong correlation between social background and educational achievement. Where should education policy focus its efforts in order to reduce these inequalities in the long term?

Educational inequalities do not arise from a single source. They are not a snapshot of a single moment in time. Rather, they develop and intensify throughout students’ entire educational journey. That is why there is no single measure that can simply eliminate them. We must intervene as early as possible while at the same time preventing initial differences from becoming increasingly pronounced as students progress through their education. That includes providing high-quality early education, the systematic development of basic language and mathematical skills, good diagnostic skills among teachers, and teaching that caters for different learning needs.

I also consider it particularly important to provide greater support to schools where the need is greatest. Schools with large numbers of pupils from disadvantaged social backgrounds require targeted – and in some cases additional – resources. Lastly, we must examine the transitions within the education system. Social inequalities arise not only because children develop different skills, but also because, even where academic performance is comparable, different educational decisions are made depending on social background. Educational equity therefore does not mean offering exactly the same to all schools and all students, but rather taking different starting points into account when providing support.

When school structures reinforce educational inequalities

One of the issues you have researched is how different school structures affect educational opportunities. What role does our tracked secondary school system play in the emergence of educational inequalities?

The tracked secondary school system is certainly not the sole cause of social inequalities in education. Differences in academic performance already exist before the transition to secondary education. However, early tracking into different school types can reinforce pre-existing differences. At least two mechanisms are at work here. First, the transition to different types of school is socially selective: Even where academic performance is comparable, the likelihood of entering particular school types differs according to children’s social background. Secondly, tracking creates different learning and developmental conditions: Students learn in differently composed groups, with different curricular requirements, expectations and learning opportunities.

The tracked secondary school system therefore not only reflects existing differences in attainment but can also influence how these differences develop over the course of students’ education. The extent to which this reinforces social inequalities also depends on how easy it is for students to move between educational pathways and what opportunities they have to change their educational path later on.

PISA primarily measures academic performance. What does the study reveal about how pupils experience school – in terms of motivation, a sense of belonging or well-being, for example?

PISA is much more than just a test of performance. The accompanying questionnaires also capture important aspects of young people’s school experience. These include their sense of belonging at school, experiences of bullying, whether young people feel safe at school and how often they are absent from school. In the field of science, the survey assesses aspects such as the extent to which pupils feel confident in tackling relevant tasks and how much they enjoy engaging with science. From an educational perspective, I consider students’ sense of belonging particularly important.

School is not only a place for acquiring skills, but also a central social environment for young people. Whether pupils feel accepted, whether they feel a sense of belonging and whether they experience social participation is therefore in itself a relevant educational outcome – and at the same time an important prerequisite for learning and developmental processes. However, we must also be cautious here in our methodological approach: PISA can highlight associations between such experiences and students’ performance, but it cannot unequivocally determine what is cause and what is effect.

By selecting what is measured and compared internationally, PISA also influences which competencies are considered particularly important. How do you assess this influence – and where should the boundary lie between international education monitoring and the way schools are organized?

There is a valid point to this criticism. Every educational study also makes normative decisions through its selection of what it measures. If reading, mathematics and science are made visible and comparable internationally, these areas inevitably receive special public and political attention. PISA thus also influences which topics are given the greatest prominence in the education debate.

The fact that PISA influences educational reforms and debates on education policy is well documented internationally. Just how strong this influence can be was seen very clearly in Germany following the first PISA study. However, I would not conclude from this that the OECD alone determines what education should be. The PISA framework concepts are developed by international expert commissions, and the participating countries are also involved in setting priorities and making key decisions.

For me, equating what can be measured in such a study with education as a whole would be where you have to draw the line. Schools have a broader educational mandate than what an international study comparing school performance can measure. For me, it is therefore important to take PISA seriously without reducing education to what PISA measures.